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This  research  was  sponsored  by  the  Office  of  the  Chief  of  Naval  Research  (Code  461)  and 
funded  as  part  of  the  Exploratory  Development  Task  entitled  Videographic  Interface  Technology 
within  the  Office  of  Naval  Research  Program  Element  0602233N,  under  die  Training  Technology 
Work  Unit  0602233N.RM33T23.03. 

The  purpose  of  this  effort  was  to  evaluate  the  use  of  technology  in  developing  and  using  a 
video-based  job  aid.  The  Engineering  Library  Visual  Information  System  (ELVIS)  used  in  this 
effort  was  developed  by  Intelligent  Systems  Design,  Inc.,  under  a  Small  Business  Innovation 
Research  (SBIR)  contract  with  the  Naval  Surface  Warfare  Center  (NSWC). 

Evaluations  of  video-based  job  aids  were  conducted  at  the  Naval  Undersea  Warfare  Center 
(NUWC)  and  the  Damage  Control  (DC)  School,  Fleet  Training  Center,  San  Diego. 

The  results  of  this  study  are  intended  for  use  in  all  Department  of  Defense  sponsored  training 
communities  and  depot  level  repair  facilities. 
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Introduction 


Summary 


Early  in  the  development  of  television,  the  medium  was  recognized  as  a  useful  educational 
tool.  Although  early  educational  television  had  a  number  of  limitations,  computer  technology  has 
allowed  television  to  be  integrated  with  a  number  of  other  media  during  individual  presentations. 
Multimedia  training  allowed  the  user  to  access  information  from  still  and  motion  video,  audio, 
graphics,  and  text  and  aided  learners  in  linking  their  knowledge  between  different  skills.  This 
combination  of  technology  provided  realistic,  interactive  training,  and  job  aids  to  individuals. 
Technology  has  not  proven  sufficient  for  many  training  development  efforts.  One  instructional  aid, 
hypermedia  provides  rapid  and  convenient  access  to  all  multimedia  links.  Hypermedia,  like  other 
instructional  techniques,  depend  on  convenient,  clear  developer  and  user  interfaces. 

This  confluence  of  domains  lead  to  the  development  of  a  Navy-sponsored  video-based 
workstation,  known  as  the  Engineering  Library  Visual  Information  System  (ELVIS).  A  prototype 
video  development,  cataloging,  and  delivery  system  was  developed  and  used  to  explore 
multimedia  development  and  implementation  issues.  ELVIS  is  intended  for  use  in  training 
procedural  tasks  that  do  not  require  developing  extensive  learning  strategies  to  guide  the  user.  To 
meet  this  goal,  ELVIS  uses  hyperlinks  to  present  combinations  of  video,  text,  and  graphic  media. 

Objective 

The  objective  of  this  effort  was  to  evaluate  ELVIS,  a  development  tool  for  producing  and 
presenting  videodisc  job  aids.  The  study  (1)  assessed  the  ability  of  novices  in  videodisc  technology 
to  develop  videodisc  job  aids  using  ELVIS  and  (2)  evaluated  the  on-the-job  and  classroom 
effectiveness  of  these  aids. 

Approach 

A  prototype  video  development  and  delivery  workstation  was  developed  under  a  Small 
Business  Innovation  Research  program  contract  This  effort  called  for  Navy  schoolhouse  and  job 
site  development  of  video-based  instruction  by  personnel  without  extensive  training  in  video 
production  skills.  The  goal  was  to  develop  job  performance  aids  that  could  be  easily  accessed  by 
instructors,  students,  or  technicians. 

The  ELVIS  workstation  is  based  on  a  Macintosh  platform  and  uses  a  write-once,  read-many 
(WORM)  optical  disc  recorder  to  capture  video.  A  separate  playback-only  device  was  used  to 
present  video. 

Software  to  produce  (author)  and  present  the  hypermedia  job  aid  was  constructed.  The  student 
could  browse  through  a  job  aid  which  provided  access  to  text,  graphics,  audio,  and  video. 
Evaluations  of  the  hypermedia  job  aids  were  conducted  at  die  Naval  Undersea  Warfare  Center 
(NUWC)  and  the  Damage  Control  (DC)  School,  Fleet  Training  Center,  San  Diego. 

Evaluation  at  NUWC  consisted  of  observing:  (1)  an  in-house  engineer  without  prior  video¬ 
development  experience  develop  a  multimedia  job  aid  for  performing  a  Mark  50  torpedo 
procedural  task  and  (2)  an  NUWC  engineer  and  an  NUWC  shop  floor  technician  use  the 
multimedia  job  aid  to  learn  to  perform  the  procedural  task. 
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At  the  DC  school,  a  detailed  multimedia  job  aid  was  developed  for  performing  maintenance  cm 
the  “Ellison”  door.  Student  performance  of  the  Ellison  door  overhaul  procedure  with  and  without 
the  job  aid  was  evaluated. 

Results 

A  NUWC  engineer  encountered  a  number  of  problems  while  constructing  a  rudimentary 
ELVIS  job  aid.  Both  NUWC  subjects  had  difficulty  performing  the  procedure  because  of 
inadequate  instructional  video  development  and  the  subjects’  inexperience  in  using  die 
hypermedia  interface. 

DC  school  student  performance  of  die  Ellison  door  overhaul  procedure  did  not  differ 
significantly  using  the  ELVIS  job  aid  than  it  did  without  using  it 

Discussion  and  Conclusions 

ELVIS  provides  a  hypermedia  template  that  allows  an  author  to  construct  a  job  aid  by  adding 
data.  This  simple  procedure  may  lull  the  developer  into  eliminating  die  requirement  for  following 
basic  instructional  development  procedures  prior  to  adding  data.  The  instructors  and  engineers 
observed  were  reluctant  to  invest  sufficient  time  for  adequate  systematic  instructional  development, 
including  video  production  or  job  aid  content  For  example,  in-house  video  development  is 
perceived  as  a  relatively  trivial  task,  but  usually  requires  specialized  skills.  In  fact  present  Navy 
policy  prevents  local  video  development  in  most  cases,  and  would  have  to  change.  Even  with 
present  video  and  hypermedia  technology,  developing  and  revising  job  performance  libraries  is  a 
time  consuming  task. 

The  Macintosh  interface  was  not  sufficiently  intuitive  for  most  of  the  subjects.  They  were  not 
able  to  interact  with  die  software  without  coaching.  DC  instructors  also  had  trouble  using  die 
interface,  even  after  moderate  practice.  Further  changes  in  technology  and  hypermedia  interfaces 
are  needed,  even  for  novice  development  and  use  of  job  performance  libraries. 

Future  Efforts 

1.  Creators  and  users  of  multimedia  job  performance  aids  should  provide  resources,  such  as  a 
software  program  to  introduce  die  mouse  interface  that  students  should  be  required  to  use,  to  guide 
die  user  in  initially  accessing  media. 

2.  Video-based  job  performance  aids  developed  for  use  in  Navy  settings  should  have  an 
in-house  change  advocate  to  ensure  adequate  development  and  use  of  the  aid. 

3.  The  constantly  changing  needs  of  Navy  training  and  task  performance  require  timely 
modification  of  all  media  used  in  job  aids.  Local  Navy  training  and  work  communities  should 
develop  in-house  skills  to  allow  continual  updates  to  computer  media. 

4.  Industry  standards  that  allow  software  and  platform  portability  should  be  monitored  to 
ensure  their  incorporation  into  future  hypermedia  job  aid  development. 
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Introduction 


Problem 

Educational  and  commercial  television  influence  us  in  many  ways.  Video  images  are  available 
to  most  people  through  broadcasts  t 1  their  home;  through  videotape  rentals;  through  group 
presentations  at  schools,  workplaces,  ami  recreational  settings;  and  in  individualized  presentations 
for  pleasure  and  instruction.  These  forms  of  video  provide  a  visual  message  that  can  be 
manipulated  to  direct  attention,  to  speed  up  or  slow  down  action,  to  view  hazardous  operations,  to 
standardize  and  disseminate  instruction,  and  to  hold  viewers’  attention.  Broadcast  television 
combines  many  of  these  desirable  instructional  attributes  at  a  reasonable  cost  Some  attributes  were 
realized  early  in  the  history  of  television  development  Closed-circuit  television  was  first  used  to 
deliver  educational  programs  in  the  early  1930s.  The  first  educational  television  broadcast  was 
presented  in  1953  at  the  University  of  Houston  (Tfexas  Higher  Education  Coordinating  Board,  1986). 

Educational  television  is  subject  to  the  same  constraints  as  any  other  presentation  having 
instructional  goals,  as  well  as  to  issues  inherent  in  televised  presentations.  Kozma  (1986)  lists  a 
number  of  variables  that  impact  on  the  effectiveness  of  television  including  pacing,  cuing,  and 
modeling.  Cuing  and  modeling  can  be  addressed  in  standard  linear  television  programming. 
However,  providing  appropriate  pacing  for  a  wide  range  of  television  viewers  is  more  difficult 
This  problem  was  addressed  successfully  by  the  advent  of  computer  technology  to  control  delivery 
of  all  media,  including  video  on  an  individual  basis. 

Multimedia  presentations  are  computer  controlled  systems  that  allow  the  user  to  access 
information  from  still  and  motion  video,  audio,  graphics,  and  text  Early  attempts  to  display  video 
on  a  computer  monitor  presented  numerous  challenges.  Videotape  was  used  to  present  motion 
visual  images,  but  random  access  was  cumbersome  and  not  reliable.  However,  a  computer  provides 
reliable  random  access  to  video  encoded  on  an  optical  disc  medium.  Students  can  easily  interact 
with  all  parts  of  the  training.  This  combination  of  computer  and  videodisc  technologies  can  be  used 
to  provide  realistic,  interactive  training  and  job  aids  to  students.  An  instructor  can  also  interact  with 
the  computer  controlling  multimedia  presentations  to  tailor  the  instruction  to  specific  classroom 
needs. 

Multimedia  instruction  may  not  be  appropriate  if  the  presentation  is  difficult  to  create  or 
students  cannot  easily  negotiate  through  the  lesson.  Other  technology  is  needed  to  ease 
development  and  use  of  multimedia  presentations.  The  idea  of  linking  textual  chunks  in  some 
meaningful  way  is  usually  credited  to  Vannevar  Bush  (1945),  who  proposed  to  mechanize  the 
storage  and  retrieval  of  scientific  literature.  Until  the  advent  of  computer  and  optical  disc  storage, 
there  was  no  easy  way  to  implement  his  proposal  for  nonlinear  or  hypertext.  Hypermedia  extends 
this  idea  to  all  forms  of  media.  The  concept  of  hypermedia  is  rapid  and  convenient  access  to 
extensive  informational  databases  that  may  take  the  form  of  text,  graphics,  video,  and  audio.  Links 
are  easily  created  between  objects  in  the  databases.  Although  hypermedia  development  has  been 
extensively  explored,  hypermedia  systems  still  depend  on  convenient  and  clear  developer  and  user 
interfaces,  which  in  many  cases  are  lacking. 

While  computer-based  interactive  multimedia  presentations  expand  the  potential  for 
procedural  training  and  job  aids,  they  also  pose  a  number  of  questions  that  have  not  been  fully 
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explored.  These  include  providing  the  correct  mix  of  media,  making  the  mix  easily  accessible  to 
students,  and  finding  ways  to  develop  affordable  multimedia  training. 

Objective 

The  objective  of  this  effort  was  to  evaluate  ELVIS,  a  development  tool  for  producing  and 
presenting  videodisc  job  aids.  The  study  (1)  assessed  the  ability  of  novices  in  videodisc  technology 
to  develop  videodisc  job  aids  using  ELVIS  and  (2)  evaluated  the  on-the-job  and  classroom 
effectiveness  of  these  aids. 

Background 

While  the  effectiveness  of  instructional  television  broadcast  to  targeted  audiences  has  produced 
some  inconclusive  findings  (Silvemail  &  Johnson,  1990),  most  studies  show  television  to  be  as 
effective  as  traditional  classroom  presentations  (Whittington,  1987).  However,  most  television 
presentations  do  not  allow  the  student  to  interact  with  the  instructional  content  for  review  and 
testing.  By  controlling  the  pacing  of  presentations,  television  is  viewed  by  some  as  a  different 
medium  from  interactive  videodisc  instruction,  where  the  student  controls  the  pacing  (Kozma, 
1986). 

Many  educators  believe  that  interactive  videodisc  instruction  (student  control  of  pacing  and 
review)  has  unique  educational  capabilities  (Beautement,  1991;  Hannafin,  1985).  To  fully  exploit 
these  capabilities,  several  basic  educational  features  must  be  incorporated  such  as  orientation 
toward  and  sequencing  of  training  materials  (Hannafin  &  Phillips,  1987).  Unfortunately,  many  of 
the  studies  investigating  interactive  videodisc  instruction  do  not  provide  sufficient  information 
about  the  use  of  these  features  to  assess  their  effects  cm  achievement  (Fletcher,  1990;  McNeill  & 
Nelson,  1991). 

Fletcher  (1990)  used  meta-analysis  techniques  to  analyze  28  studies  that  compare  interactive 
videodisc  instruction  with  conventional  instruction  in  military,  industrial,  and  college  settings.  In 
studies  that  report  training  time,  Fletcher  found  a  moaest  increase  in  performance  and  a  31% 
reduction  in  training  time.  However,  the  interactiveness  of  this  training  medium  may  lead  students 
to  increase  time  on  task. 

Cennamo,  Savenye,  and  Smith  (1991)  found  the  amount  of  time  students  needed  to  complete 
a  lesson  differed  significantly  between  three  treatment  groups.  A  technical  science  lesson  was 
presented  via  linear  television,  instructional  television,  or  interactive  video.  Students  watched  die 
linear  presentation  passively,  while  instructional  television  and  interactive  video  requited  them  to 
respond  to  questions.  Students  viewing  die  interactive  presentation  could  view  feedback  as  long  as 
they  desired,  whereas  the  instructional  television  presentation  was  group-paced  and  presented  the 
same  feedback  for  a  fixed  time.  The  mean  time  students  needed  to  complete  the  interactive  video 
lesson  was  25.3  minutes,  the  instructional  television  lesson  was  23.2,  and  the  linear  television 
lesson  was  18.5  minutes.  Recall  scores  for  students  who  used  interactive  video  were  significantly 
higher  than  for  those  who  watched  linear  television,  but  there  was  no  difference  between  the  scores 
of  students  who  used  the  interactive  video  and  those  who  used  instructional  television. 

While  interactive  videodisc  instruction  has  been  shown  to  provide  significant  achievement 
effects  (Fletcher,  1990),  the  development  costs  for  this  type  of  instruction  can  be  high.  Most  studies 
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of  interactive  videodisc  instruction  do  not  address  the  development  costs  faced  in  video 
production.  Beautement  (1991)  estimates  production  of  a  single  1-hour  videodisc  costs  $380,000 
in  die  United  Kingdom.  This  cost  may  be  acceptable  compared  to  the  cost  of  developing  highly 
realistic  simulators,  but  many  job  training  environments  cannot  tolerate  this  expense.  They  need 
an  in-house  capability  for  development  of  technologically  advanced  video  presentations  designed 
by  schoolhouse  personnel,  and  controlled  by  instructors,  students,  or  on-the-job  trainees. 

Navy  Interactive  Videodisc  Draining  Investigation 

The  Fleet  Engineering  Special  Projects  Office  at  the  Naval  Surface  Weapons  Center 
Detachment,  White  Oak  Laboratory,  Silver  Spring,  MD  became  interested  in  interactive  videodisc 
job  aids  in  the  early  1980s  with  the  focus  on  maintenance  and  repair  of  weapons  systems.  The 
initial  objective  was  to  provide  a  means  for  producing  inexpensive  interactive  videodisc  job  aids 
and  training  in  a  depot  maintenance  setting.  This  objective  eliminates  professional  video 
production  while  empowering  schoolhouse  and  shop  personnel  to  develop  and  produce  interactive 
video  training.  A  Small  Business  Innovation  Research  (SBIR)1  contract  was  awarded  to 
investigate  the  feasibility  of  using  nonexperts  to  produce  and  transfer  video  to  a  microcomputer 
workstation  and  to  use  production  tools  to  author  job/instructional  aids. 

This  SBIR  research  resulted  in  the  development  of  a  videodisc  workstation,  known  as  the 
Engineering  Library  Visual  Information  System  (ELVIS).  ELVIS  is  intended  for  use  in  training 
procedural  tasks  that  do  not  require  development  of  extensive  learning  strategies  to  guide  the  user. 
To  meet  this  goal,  ELVIS  uses  hyperlinks  for  presenting  combinations  of  video,  text,  and  graphic 
media.  This  prototype  video  development,  cataloging,  and  delivery  system  was  used  in  the  effort 
reported  here  to  explore  video  development  and  implementation  issues. 

Approach 

Video  presentations  have  been  incorporated  in  a  variety  of  Navy  training  programs,  from 
traditional  group-paced  classroom  and  shipboard  presentations  to  computer-controlled, 
individualized  videodisc  presentations  in  schoolhouse  laboratories  or  at  shore-based  job  sites. 
However,  most  video  applications  depend  upon  development  efforts  by  professionals  not  at  the 
Navy  schoolhouse  or  job  site.  This  effort  called  for  in-house  development  of  video-based 
instruction  by  personnel  without  training  in  video  production  skills.  The  ultimate  goal  was  to 
develop  low  cost,  easily  modified  job  performance  aids  that  provide  easy-to-use  procedural  guides. 
During  phase  I  of  the  SBIR  effort,  a  prototype  video  workstation  was  developed.  Phase  n  research 
refined  this  development  and  delivery  workstation  to  include  interactive  videodisc. 

Initial  evaluation  of  this  workstation  was  undertaken  by  the  Naval  Undersea  Warfare  Center 
(NUWC),  Division  Keyport,  and  the  Navy  Personnel  Research  and  Development  Center 
(NAVPERSRANDCEN).  The  NUWC  evaluation  consisted  of  an  in-house  engineer  without  video 
development  experience  developing  a  videodisc  job  aid  for  a  Mark  50  torpedo  procedural  task.  A 
NUWC  engineer  and  a  shop  floor  technician  used  this  job  aid  to  perform  the  task. 

1This  SBIR  work  was  performed  by  Intelligent  Systems  Design  (ISD),  Inc.,  16771  N.  E.  80th  Street,  Suite  101, 
Redmond,  WA  98052  under  contract  with  the  Naval  Surface  Weapons  Center  Detachment,  a  branch  of  the  Naval 
Surface  Warfare  Center. 


A  second  evaluation  was  carried  out  at  the  Damage  Control  (DC)  School,  Fleet  Training 
Center,  San  Diego,  where  development  of  interactive  videodisc  job  aids  for  several  damage  control 
tasks  was  jointly  undertaken  by  NAVPERSRANDCEN  personnel  and  DC  instructors.  Student 
performance  was  assessed  following  instruction  with  and  without  video  job  aids. 

ELVIS  Workstation 

To  provide  a  workstation  that  would  be  easy  to  use  to  develop  and  display  multimedia 
instruction  in  die  form  of  still  and  motion  video,  audio,  text  and  graphics  in  an  interactive  setting, 
Patton,  Staley,  &  Chin  (undated)  stated  that  the  system  should: 

1.  Be  similar  in  functionality  to  personal  computer-based,  desk-top  publishing  (DTP)  systems. 

2.  Be  useable  by  technicians  who  have  no  special  training  in  computer,  video,  and  multimedia 
technologies. 

3.  Be  able  to  collect,  add,  and  update  text,  graphics,  still  and  motion  video,  and  sound  data  in 
a  multimedia  database. 

4.  Be  based  on  the  Apple  Macintosh  O2  platform  (to  be  compatible  with  an  existing  NUWC 
logistics  system),  and  include  video  capture  and  processing  cards,  write-once  optical  discs  to  store 
analog  video,  and  a  large  capacity  hard  disc  drive. 

3.  Be  capable  of  organizing,  storing,  searching,  linking,  and  retrieving  job  tasks  which  have 
been  attached  to  text,  graphics,  still  and  motion  video,  and  sound  resources. 

6.  Contain  all  die  forms  of  media  in  a  job  task  aid  so  that  interactive  information  describing  that 
job  task  can  be  accessed  by  new  and  existing  operators.  The  job  task  aid  will  be  used  to  test  and 
evaluate  the  ease  of  use  and  the  resulting  quality  of  an  interactive  program  authored  in  ELVIS. 

To  meet  these  requirements  required  making  a  number  of  compromises  in  selecting  both 
hardware  and  software.  Since  these  selections  were  made  (in  1990  and  early  1991),  the  technology 
has  continued  to  evolve.  ELVIS  workstation  hardware  and  software  are  described  in  the  appendix. 

ELVIS  development  included  evaluation  of  several  video  cameras  and  recorders  (Patton  et  al., 
undated).  The  criteria  for  evaluating  this  equipment  were:  (1)  image  resolution,  (2)  ease  of  use, 
(3)  ruggedness,  and  (4)  cost.  S-VHS  tape  format  was  selected  as  having  the  best  resolution  for  the 
cost.  A  single  unit  camcorder  offered  the  best  tradeoff  between  portability  and  operator  fatigue.  To 
avoid  fatigue,  Patton,  et  al.  (undated)  recommended  that  the  camera  operator  limit  taping  to  a  few 
minutes  at  a  time  and  use  a  tripod  when  possible.  Placing  a  microphone  on  the  person  speaking  was 
recommended  for  obtaining  audio  because  camera  mounted  microphones  are  unsuitable  for  use  in 
noisy  environments.  To  aid  inexperienced  personnel,  a  cart  with  two  monitors  (one  facing  the 
camera  operator  and  the  other  facing  the  on-camera  speaker)  was  recommended. 


identification  of  software  and  specific  equipment  is  for  documentation  only  and  does  not  imply  endorsement 


Mark  50  Torpedo  Maintenance  Job  Performance  Aid 

How  technology  can  aid  a  novice  in  video  development  and  job  aiding  was  addressed  first. 
NUWC,  which  serves  as  a  primary  undersea  weapon  test,  evaluation,  engineering  agent,  and  repair 
depot,  is  a  highly  automated  facility  that  uses  Macintosh  computers  to  monitor  and  control  torpedo 
repair  logistics.  Many  repair  tasks  are  complicated  and  are  not  routinely  performed.  Technicians 
performing  these  tasks  can  benefit  from  interactive  video  job  aids. 

Development 

To  understand  the  video  development  process,  we  observed  an  NUWC  engineer  (Engineer  1) 
develop  a  job  aid.  Engineer  1  was  familiar  with  the  Macintosh  operating  system  used  in  ELVIS. 
However,  she  had  no  experience  with  developing  a  storyboard,  shooting  video,  or  other  production 
techniques  used  in  creating  the  job  performance  aid. 

On  die  day  prior  to  shooting  the  video,  Engineer  1  developed  a  short  procedural  job  aid  “script” 
to  serve  as  a  storyboard.  The  script  indicated  which  parts  of  the  procedure  would  be  still  images 
and  which  would  be  action  sequences.  The  task  was  to  make  a  flow  adjustment  in  an  oxidizer 
(commonly  called  the  oxystock)  unit  that  is  part  of  the  Mark  SO  torpedo  propulsion  unit  The  script 
divided  the  task  into  greater  detail  than  the  description  of  the  procedure.  Using  die  script, 
Engineer  1  reviewed  the  procedure  with  the  technician  who  would  appear  on  camera. 

Prior  to  video  production.  Engineer  1  viewed  a  training  aid  created  by  the  developer  of  ELVIS 
(ISD,  Inc.)  to  gain  insight  into  the  video  development  process  This  training  aid  provided  an 
overview  of  camera  operation,  lighting,  and  production  techniques. 

Following  this  viewing,  die  video  was  produced.  The  technician  performed  and  narrated  die 
actual  on-camera  work.  Another  NUWC  engineer  (Engineer  2),  who  had  previously  developed  an 
ELVIS-based  job  aid,  assisted  with  video  cabling  and  did  the  actual  camera  work,  which  was 
directed  by  Engineer  1.  Approximately  5  minutes  of  videotape  were  produced.  This  unedited 
videotape  was  transferred  to  a  write-once,  read-many  (WORM)  disc,  indexed  by  initial  and  end 
frames  of  each  job  sequence,  and  then  linked  to  specific  job  structure  steps.  Engineer  1  transferred 
a  previously  prepared  text  file  to  the  ELVIS  editor  to  complete  development  of  the  job  aid  (with 
die  help  of  a  systems  analyst  from  die  ELVIS  contractor  because  of  software  problems).  At  the  end 
of  the  day,  the  job  performance  aid  was  complete.  Some  problems  that  were  corrected  over  die  next 
few  days  were  noted  in  review.  On  the  day  following  initial  development.  Engineer  1  was 
interviewed  about  the  development  process. 

Trials 

Two  NUWC  employees  were  randomly  selected  to  perform  the  Mark  50  torpedo  oxystock 
adjustment  using  the  job  performance  aid.  Neither  employee  was  familiar  with  the  procedure. 
Subject  1  was  an  electrical  engineer  who  had  not  worked  with  the  Mark  50  torpedo.  He  was 
familiar  with  the  Macintosh  computer,  computer-based  training  systems  in  general,  and  had 
experience  with  video  production.  Subject  2  was  a  Mark  46  technician,  who  was  not  familiar  with 
either  the  Mark  50  torpedo  or  the  oxystock  adjustment  procedure  and  had  never  used  a  personal 
computer,  or  a  home  video  play  or  record  device. 
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Prior  to  the  trials.  Engineer  2  made  some  minor  revisions  in  the  job  aid  text.  Additional  changes 
were  made  in  both  the  text  and  video  portions  of  the  job  aid  based  on  die  observations  of  the  first 
trial.  Both  trials  were  videotaped  to  document  each  subject’s  performance  using  the  job  aid. 

Engineer  2  briefed  both  subjects  on  the  purpose  of  the  clinical  trial  and  the  task  they  were  to 
perform.  Since  Subject  1  was  familiar  with  the  Macintosh  computer,  common  features  of  the  user 
interface  such  as  die  mouse,  windows,  icons,  and  scroll  bars  were  explained  only  to  Subject  2. 
A  short  orientation  to  the  ELVIS  user  interface  was  given  to  both  subjects.  This  included  selecting 
job  task  steps  and  substeps  using  the  ELVIS  hierarchical  database  structure,  replaying  associated 
video  information,  and  expanding  graphic  displays.  The  orientation  was  not  scripted  beforehand. 

Ellison  Door  Maintenance  Job  Performance  Aid 

NAVPERSRANDCEN  sought  out  a  Navy  schoolhouse  site  where  an  interactive  video  training 
aid  could  be  used  as  an  adjunct  to  classroom  and  laboratory  instruction.  The  DC  school  was  already 
participating  in  NAVPERSRANDCEN  research  involving  classroom  distance  learning  using 
remote  television  broadcasts.  Because  some  tasks  for  the  distance  learning  required  observation  of 
hands-on  procedures,  it  was  decided  to  produce  a  set  of  video  job  aids  to  enhance  laboratory 
exercises  following  remote  broadcasts.  Six  relatively  short  procedures  were  selected. 
NAVPERSRANDCEN  provided  the  video  equipment  and  expertise,  while  the  DC  school  provided 
scripts  and  on-camera  procedural  experts.  The  job  structures  were  entered  into  die  job  aid  windows 
and  linked  to  die  videodisc  to  create  six  job  performance  aids.  These  job  aids  were  shown  in  the 
classroom  on  35-inch  monitors.  The  instructor  could  rapidly  display  any  procedure  or  repeat  a 
specific  sequence  by  selecting  an  ELVIS  job  structure.  However,  students  could  not  use  ELVIS  in 
the  laboratory  setting. 

Development 

Another  task  was  sought  to  provide  an  evaluation  of  job  performance  aid  development  and 
student  and  instructor  interaction  with  ELVIS.  Discussion  with  DC  school  instructors  resulted  in 
die  selection  of  maintenance  procedures  for  the  machinery  space  balanced  joiner  door,  known  as 
die  Ellison  door. 

Initial  attempts  to  have  the  DC  instructors  produce  the  necessary  video  were  unsuccessful.  The 
instructors  either  lacked  confidence  in  their  ability  to  produce  useable  video  for  training  or  were 
not  willing  to  devote  the  time  required  for  producing  video  while  maintaining  normal  teaching 
duties. 

Consequently,  NAVPERSRANDCEN  and  the  DC  school  agreed  to  a  joint  video  development 
effort:  NAVPERSRANDCEN  provided  video  experts  while  the  DC  school  provided  detailed 
procedural  tasks  that  could  be  roughly  translated  into  a  working  script  One  or  two  DC  school 
instructors  performed  the  procedures,  which  were  videotaped.  NAVPERSRANDCEN  personnel 
used  professional  editing  equipment  to  produce  1  hour  of  motion  video  and  stills  from  4  hours  of 
videotape.  The  tape  was  shot  over  a  2-day  period.  Verbal  procedures  were  added  to  the  tape  by  a 
DC  school  instructor  reading  a  script  This  tape  was  transfered  to  a  WORM  optical  disc,  which 
required  no  further  editing.  Development  of  the  final  ELVIS  job  performance  aid  took  1  month. 
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Evaluation 


An  ELVIS  workstation  on  a  movable  cart  was  maintained  in  the  instructors’  office  space  and 
moved  to  the  classroom  or  the  Ellison  door  laboratory  as  needed.  Instructors  were  briefed  on  the 
use  of  the  system  and  were  requested  to  instruct  students  on  its  operation.  Instructors  or  students 
used  die  mouse  interface  to  select  Ellison  door  maintenance  procedures.  A  seven-step  procedural 
outline  to  power  up  die  Macintosh  was  taped  to  the  cart 

The  Ellison  door  maintenance  course  is  taught  once  every  2  weeks;  it  lasts  approximately 
2  days,  lb  familiarize  the  instructors  in  operation  of  ELVIS  and  to  observe  student  attitudes  toward 
die  job  aid,  instructors  for  the  two  classes  in  My  1992  used  ELVIS  in  both  the  classroom  and 
laboratory.  In  the  classroom,  instructors  manipulated  die  ELVIS  interface  to  select  video  sequences 
for  student  viewing.  In  the  laboratory,  students  were  encouraged  to  use  the  Ellison  maintenance 
performance  aids  at  appropriate  times. 

Because  of  limited  classroom  and  laboratory  spaces,  and  small  class  size,  it  was  not  possible 
to  assign  students  from  the  same  class  randomly  to  ELVIS  or  conventional  training  conditions. 
During  the  next  2  months,  the  first  class  of  the  month  was  shown  the  ELVIS  job  aids  in  the 
classroom  and  asked  to  use  the  job  aids  in  support  of  the  laboratory  procedures.  The  second  class 
of  die  month  was  instructed  in  the  conventional  way;  dial  is,  classroom  students  viewed  a  video  for 
civilian  Ellison  door  installers,  and  used  a  combination  of  worksheet  and  instructor  supervision  in 
the  laboratory.  The  class  size  during  this  period  ranged  from  three  to  nine  students. 

Following  die  laboratory,  students  were  tested  on  a  critical  component  of  the  Ellison 
Model  139B  door  maintenance  procedures:  overhauling  the  hydraulic  cylinder.  The  same  observer 
monitored  all  the  testing  using  a  checklist  to  ensure  reliability  across  students.  Because  of  the  time 
required  for  testing  and  the  availability  of  only  one  observer,  not  all  students  were  tested.  The  first 
class  consisted  of  three  students,  all  of  whom  were  tested.  In  other  classes,  three  students  were 
randomly  selected  to  represent  the  same  levels  of  experience  found  in  the  first  class.  This  resulted 
in  a  total  of  six  students  tested  under  each  condition.  Measures  were  collected  on  overhaul 
performance,  safety,  time  on  task,  and  the  standard  DC  school  written  final  examination. 

The  performance  measure  consisted  of  disassembly,  inspection,  and  reassembly  of  an  Ellison 
door  hydraulic  cylinder.  The  observer  scored  each  step  as  it  was  completed.  Four  of  the  steps 
required  that  the  student  take  appropriate  safety  precautions.  Any  student  who  did  not  take  them 
was  stopped  and  advised  to  do  so  by  the  observer  prior  to  continuing.  Each  student  was  videotaped 
for  later  review  to  clarify  any  discrepancies.  All  students  took  die  same  25-item  multiple-choice 
written  test  at  the  end  of  the  laboratory  session. 

Results 

Mark  50  Torpedo  Maintenance  Job  Performance  Aid 

Development 

Engineer  1  used  computers  extensively  on  the  job,  and  perceived  herself  to  be  a  fairly 
knowledgeable  computer  user.  She  felt  that  technology  had  a  positive  impact  on  both  her  home  and 
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work  situations.  In  general.  Engineer  1  found  the  ELVIS  interfaces  straightforward,  although  she 
considered  the  task  structure  editing  process  poorly  designed.  Using  a  version  of  the  software  that 
had  not  been  completely  debugged  created  other  problems.  With  the  aid  of  Engineer  2  and  a 
contractor’s  representative,  a  rudimentary  job  aid  that  cou’i  be  used  for  evaluating  user 
performance  was  produced.  The  job  aid  had  minimum  revision  prior  to  the  trials. 

Trials 

Subject  1  performed  die  majority  of  disassembly  and  reassembly  steps  with  only  one  viewing 
of  the  NUWC  job  aid,  and  displayed  considerable  confidence  during  most  of  the  steps.  He  watched 
and  performed  the  procedure  one  step  at  a  time,  without  previewing  the  entire  procedure  prior  to 
performance.  During  the  trial  it  became  apparent  that  Subject  1  did  not  completely  understand  how 
to  access  some  job  aid  steps.  Engineer  2  had  to  intervene  on  several  occasions  to  provide 
clarification. 

Subject  1  was  able  to  perform  the  majority  of  steps  accurately  with  some  exceptions.  Because 
the  job  aid  was  created  hurriedly  and  without  any  revision,  some  of  its  shortcomings  became 
apparent  when  technicians  with  no  knowledge  of  the  Mark  50  torpedo  attempted  to  use  the  job  aid. 

1.  When  Subject  1  reached  die  key  step  of  the  procedure,  the  actual  adjustment  of  the  flow 
valve,  he  expressed  confusion  over  what  should  be  done,  reviewed  the  associated  video  segments 
several  times,  and  displayed  uncertainty  as  to  whether  he  was  performing  die  adjustment  correctly. 

2.  At  the  end  of  the  procedure,  still  video  images  replaced  instances  of  full  motion  video. 
Initially,  Subject  1  hesitated  to  perform  steps  supported  only  by  still  video  images,  although  they 
were  adequate  for  understanding  the  step.  Subject  1  eventually  performed  these  steps,  but  with  less 
confidence  than  the  steps  supported  by  full  modem  video.  This  was  observed  in  the  subject’s  visual 
fixation;  he  continually  alternated  between  looking  at  the  still  visual  and  the  hardware  before 
performing  a  step.  During  debriefing,  the  subject  confirmed  that  he  was  confused  by  die  sudden 
change  in  the  presentation  of  the  visual  material.  He  indicated  that  he  had  come  to  rely  on  viewing 
die  motion  video  and  felt  deprived  when  it  was  not  provided. 

Subject  2  was  very  uncomfortable  and  nervous  at  the  start  of  his  trial  and  had  considerable 
difficulty  mastering  the  mouse.  He  had  difficulty  correlating  cursor  position  with  movement  of  the 
mouse  and  was  unable  to  move  the  cursor  on  the  smaller  icons  and  control  boxes  on  the  screen. 
Windows,  “point  and  click”  interface,  icons,  and  scrolling  were  new  to  him.  He  appeared  to  be  very 
frustrated;  that  is,  he  sat  staring  at  the  screen  until  the  engineer  provided  additional  instruction. 
Subject  2’s  orientation  to  ELVIS  job  structure  was  equally  difficult  He  did  not  grasp  the  concept 
of  the  hierarchical  presentation  of  job  aid  text  He  also  had  difficulty  using  the  video  remote  control 
unit  that  was  operated  through  the  mouse. 

Once  Subject  2  felt  comfortable  with  the  mouse  interface  and  could  open  the  job  aid  windows, 
he  proceeded  with  die  task,  but  at  a  somewhat  lower  performance  level  than  Subject  1.  As 
Subject  2  grew  more  confident  he  performed  each  step  noticeably  faster.  He  also  exhibited  a 
dramatic  learning  curve  with  the  mouse  and  became  proficient  at  replaying  the  video  information. 
Several  problems  were  observed,  however. 
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1.  The  organization  of  the  job  procedure  text  in  a  hierarchical  structure  continued  to  confuse 
Subject  2;  when  he  reached  the  bottom  of  a  subtask,  he  needed  assistance  to  move  up  the  hierarchy 
to  the  next  level. 

2.  Subject  2  did  not  use  the  auxiliary  text  window  that  displays  special  additional  information 
and  warnings  because  ELVIS  provides  no  cue  to  look  at  the  window  when  a  new  message  is 
displayed. 

3.  On  several  occasions,  when  a  video  segment  was  available,  he  did  not  read  the  associated 
job  aid  text  or  use  graphic  displays.  In  many  cases,  die  video  did  not  provide  sufficient  information 
to  perform  a  step,  especially  toward  die  aid  of  the  procedure  when  still  video  images  were  used. 

Subject  2  exhibited  behavior  similar  to  that  of  Subject  1.  He  shifted  his  attention  between  the 
visual  images  on  the  monitor  and  the  hardware  several  times  before  performing  most  steps,  and 
exhibited  the  same  hesitation  and  confusion  over  die  key  flow  valve  adjustment  step  as  Subject  1. 
He  was  able  to  perform  the  adjustment  only  with  direct  intervention  by  die  engineer.  The  text  and 
visual  information  presented  for  this  step  were  obviously  inadequate  for  performance  by  a  novice. 

Ellison  Door  Maintenance  Job  Performance  Aid 

A  series  of  one-way  analyses  of  variance  (ANOVAs)  was  computed  for  performance  score, 
safety  score,  aid  performance  time  for  the  Ellison  door  hydraulic  cylinder  overhaul  procedure;  and 
final  written  test  covering  all  course  content.  Tfcble  1  presents  the  mean  and  standard  deviation  far 
each  of  die  dependent  variables;  and  Tfeble  2,  the  results  of  the  ANOVA  for  each  variable. 

Tkblel 

Performance  Score  Means  and  Standard  Deviations 
for  Students  Using  a  Job  Performance  Aid  (Group  1) 
or  Conventional  Instruction  (Group  2) 


Dependent  Variable 

Group 

Mean 

Standard 

Deviation 

Overhaul  Performance  Score 

1 

19.2 

33 

2 

21.5 

1.8 

Performance  Safety  Score 

1 

35 

0.8 

2 

3  JO 

0.9 

Performance  Time  (minutes) 

1 

16.0 

1.9 

2 

16.2 

1.1 

Final  Written  Tfest  Score 

1 

84.0 

13.1 

2 

90.7 

5.4 

Ifcble  2  shows  there  were  no  significant  differences  between  die  two  groups  mi  any  of  die 
measures.  Table  1  shows  that  the  group  using  the  ELVIS  job  aid  (Group  1)  made  more  mors  on 
die  hydraulic  cylinder  overhaul  and  the  final  written  test,  while  die  group  receiving  conventional 
instruction  (Group  2)  made  more  safety  errors  and  took  slightly  longer  to  perform  the  overhaul. 
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Table  2 


Results  of  Analyses  of  Variance 


Source 

df 

MS 

F 

Performance  Within  Groups 

1 

1633 

232 

10 

7.03 

Safety  Within  Groups 

1 

0.75 

1.00 

10 

0.75 

Time  Within  Groups 

1 

0.08 

0.006 

10 

14.06 

Written  Tfest  Within  Groups 

1 

13333 

132 

10 

101.33 

Note.  No  significant  differences  were  found. 

Discussion 

Mark  50  Torpedo  Video  Job  Aid  Development  and  Use 

The  addition  of  media  such  as  still  and  motion  video,  audio,  and  graphics  to  augment  traditional 
computer  text  displays  is  occurring  rapidly.  ELVIS  is  (me  example  of  this  phenomenon.  The  results 
of  the  Marie  50  job  performance  aid  exposed  a  number  of  problems  associated  with  integrating  an 
interactive  multimedia  system  into  Navy  instruction.  The  same  rules  that  apply  to  basic 
instructional  development  apply  just  as  strongly  to  interactive  multimedia  job  aids. 

The  results  of  the  rapid  development  of  the  NUWC  video  job  aid  illustrate  some  of  these 
implementation  problems.  The  oxystock  job  aid  was  developed  over  2  days  and  not  subjected  to 
formative  evaluation  prior  to  our  observation  of  its  use.  As  the  trials  showed,  rapid  development 
must  be  followed  by  further  work  to  sort  out  die  development  bugs.  For  example,  the  change  finom 
full-motion  video  to  still  video  and  graphic  images  had  been  done  to  evaluate  die  ease  of 
developing  these  media,  not  because  it  was  considered  the  best  way  to  portray  these  steps.  Both 
subjects  performed  the  early  steps  of  the  procedure  better  when  viewing  motion  video.  Studies  on 
procedural  learning  using  motion-based  aids  indicate  that  motion  is  usually  beneficial  when  it  aids 
in  task  discrimination,  especially  when  combined  with  hands-on  practice  (Wetzel,  Radtke,  &  Stem, 
1993).  However,  each  training  situation  must  be  evaluated  separately  because  costs  can  be  lowered 
dramatically  if  stills  can  meet  training  needs. 

An  interactive  multimedia  job  aid  can  be  a  powerful  and  effective  way  to  guide  a  user  in 
performing  a  procedure,  if  the  procedure  has  been  developed  into  a  clear  presentation.  When 
full-motion  video  adequately  supported  the  procedural  steps,  the  Mark  50  job  aid  virtually 
eliminated  any  ambiguity  about  what  was  to  be  done  or  how  it  was  to  be  performed.  However,  in 
several  instances  the  subjects  had  difficulty  understanding  a  step  because  the  video  or  graphic 
material  was  not  clear  or  because  there  were  breaks  in  continuity.  These  parts  of  the  job  aid 
assumed  a  prior  level  of  knowledge  on  the  part  of  the  technician  that  would  not  support  a  novice. 
Although  professional  video  presentations  in  terms  of  lighting,  color,  and  sound  quality  may  not 
be  necessary,  the  visual  information  in  the  job  aid  must  reflect  basic  principles  regarding  content 
consistency,  continuity,  and  step  size  and  document  critical  steps  properly. 


The  two  subjects  are  at  opposite  ends  of  the  continuum  representing  use  and  understanding  of 
basic  computer  functions.  Subject  1  (the  engineer)  had  prior  Macintosh  experience  and  was 
comfortable  manipulating  the  mouse.  In  contrast.  Subject  2  (the  technician)  displayed  a  total  lack 
of  confidence  in  his  ability  to  use  the  computer  and  required  considerable  time  to  acclimate  himself 
before  initiating  the  procedure.  Both  subjects  had  trouble  initially  using  the  ELVIS  hypermedia. 
The  immediate  use  of  a  multimedia  job  aid  by  a  totally  naive  user  appears  unlikely  given  the 
current  popular  interface  technology. 

Ellison  Door  Maintenance  Job  Performance  Aid  Development  and  Use 

Development  of  the  Ellison  door  maintenance  job  aid  illustrates  many  of  the  problems  faced 
when  introducing  new  technology  to  a  schoolhouse  setting.  Even  though  DC  school  instructors 
were  interested  in  incorporating  ELVIS  hypermedia  job  aids  into  die  Ellison  door  maintenance 
curriculum,  die  video  and  computer  technology  intimidated  them.  Job  aid  development  required 
extensive  NAVPERSRANDCEN  support.  Instructors  were  reluctant  to  learn  proper  operating 
procedures  for  the  Macintosh  and  ELVIS  software.  They  attempted  to  start  ELVIS  through  trial  and 
error  rather  than  refer  to  the  simple  seven-step  start  up  procedure  taped  to  the  ELVIS  cart 
Similarly,  they  did  not  follow  instructions  for  turning  ELVIS  off,  which  caused  occasional  software 
failures. 

Some  of  the  problems  were  caused  by  die  instructors’  lack  of  computer  literacy.  Three  of  die 
four  instructors  had  never  used  a  mouse  before  and  me  had  no  prior  computer  experience.  The 
instructors  made  a  number  of  errors  manipulating  ELVIS  media  from  which  they  needed  help  to 
recover.  As  with  the  NUWC  technician,  the  instructor  without  computer  experience  had  difficulty 
with  the  mouse. 

The  two  instructors  who  participated  in  development  of  the  job  aid  were  instructors  for  the 
experimental  classes.  Even  though  they  had  practiced  with  the  Macintosh  and  ELVIS  software, 
they  still  required  considerable  help  in  accessing  job  aid  steps.  Because  of  their  involvement  in 
development,  we  hoped  they  would  be  strong  advocates  of  die  use  of  this  technology  in  the 
classroom  and  laboratory.  Although  these  instructors  encouraged  students  to  use  ELVIS  in  the 
laboratory,  few  students  took  advantage  of  the  video  segments  even  though  they  were  told  that 
ELVIS  would  substitute  for  the  normal  workbook.  The  instructors  were  reluctant  to  require  the 
students  to  use  the  ELVIS  job  aid. 

It  is  not  clear  whether  students  were  afraid  to  use  the  mouse  interface  or  just  wanted  to  finish 
die  laboratory  as  soon  as  possible.  Some  students  stood  in  front  of  the  monitor  looking  at  the  text 
and  video  stills,  but  very  few  attempted  to  manipulate  the  mouse  to  get  motion  video,  even  when 
NAVPERSRANDCEN  observers  encouraged  the  students  to  access  the  information.  Students  gave 
a  number  of  reasons  for  not  using  ELVIS;  however,  no  pattern  emerged  that  would  explain  their 
reluctance  to  use  the  job  aid  in  the  laboratory  setting. 

Revising  a  Job  Performance  Akl 

Although  ELVIS  was  supposed  to  provide  convenient  provisions  for  revising  text,  graphics, 
and  motion  video,  this  design  parameter  was  not  met.  It  is  unlikely  that  a  video-based  job  aid  can 
be  constructed  without  some  form  of  formative  evaluation  and  revision  to  ensure  all  users  can 
perform  die  procedure  using  only  the  aid.  Although  this  is  particularly  true  when  the  job  aid 
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developer  is  a  relative  novice,  no  instructional  developer  is  likely  to  make  revisions  if  making  diem 
is  perceived  as  an  onerous  or  time  consuming  task.  This  problem  is  pandemic  to  the  field  of 
computer  software.  While  software  is,  in  most  cases,  described  as  easy  to  use,  in  many  cases 
extensive  efforts  are  needed  by  the  user  to  gain  proficiency. 

Computer  Orientation  for  the  Novice 

The  Macintosh  mouse-based  interface  was  not  sufficiently  intuitive  for  the  users,  who  were  not 
able  to  interact  with  the  software  without  coaching.  Most  students  were  not  familiar  with  the 
Macintosh  or  any  other  computer  system.  Although  students  claimed  they  did  not  feel  threatened 
by  technological  advances,  few  willingly  approached  ELVIS  in  the  laboratory,  and  only  one 
attempted  to  explore  the  job  aid.  Instructors  using  the  ELVIS  job  aid  appeared  confused  over  how 
to  perform  simple  functions,  such  as  moving  from  lower  to  higher  levels  in  the  job  structure 
hierarchy. 

Some  of  these  problems  may  be  overcome  by  use  of  alternative  interfaces.  The  most  logical 
alternative  is  a  touch  screen,  which  novice  computer  users  may  accept  more  readily.  However,  a 
touch  screen  would  require  some  trade-offs.  Resolution  of  selection  points  may  be  decreased.  Cost 
would  increase  and  reliability  may  suffer.  In  many  Navy  environments,  students  work  with  fluids, 
solvents,  and  greases  that  may  damage  the  touch  screen. 

Alternatives  to  ELVIS:  Other  Instructional  Hypermedia 

Computer  literacy  presently  is  not  a  prerequisite  for  DC  classes  or  most  other  Navy  training. 
While  new  technology  is  continually  changing  both  classroom  and  shipboard  environments, 
students  and  shipboard  personnel  often  are  not  prepared  to  take  full  advantage  of  these 
technological  advances.  We  observed  many  ELVIS  users  who  were  unwilling  or  unable  to  progress 
through  a  job  aid.  If  users  feel  helpless,  they  are  unlikely  to  continue  attempting  to  access  the  job 
aid  information,  no  matter  how  useful  the  information  may  be.  Technological  and  human  factor 
enhancements  of  current  interfaces  are  needed  to  increase  the  probability  that  developers  and  users 
will  use  hypermedia  workstation  features.  Several  commercial  products  offer  partial  solutions  to 
some  of  these  problems. 

One  of  the  most  promising  approaches  is  the  concept  of  a  hypermedia  environment  enhanced 
by  other  software  extensions  and  tools.  While  ELVIS  is  based  on  some  hypermedia  principles, 
many  computer  novices  found  the  technology  for  implementing  these  principles  difficult  to  use. 
Some  commercial  hypermedia  systems  offer  die  potential  of  simplified  authoring,  powerful  search 
strategies  of  the  database,  and  other  enhancements.  At  this  time,  there  are  systems  that  operate 
across  microcomputer  platforms3  and  others  that  have  the  potential  for  translation  across 
platforms.4  Of  these  software  programs,  HyperCard  has  a  relatively  long  history.  HyperCard, 
originally  developed  by  Apple  Computer,  Inc.,  resulted  in  the  development  of  a  large  base  of 


3Examples  of  these  commercial  hypermedia  products  are  Authorware  Pro  (Macromedia  Inc.,  approximate  cost  of 
$8,000)  and  MacroMind  Director,  which  lacks  some  authoring  capabilities  (Macromedia  Inc.,  approximate  cost  of 
$1,195). 

'Examples  include  systems  in  Macintosh  format  such  as  HyperCard  (Claris  Coro.,  approximate  cost  of  $199)  or 
SupetCard  (Aldus  Corp.,  approximate  cost  of  $299)  and  systems  in  Windows  format  such  as  Guide  3.1  (Owl 
International,  cost  approximately  $795)  or  ToolBook  (Asymetrix  Corp.,  approximate  cost  of  $395-695). 
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external  commands,  which  allows  the  HyperCard  scripting  language  to  be  used  with  a  wide  variety 
of  hardware  components. 

The  ELVIS  hypermedia  system  uses  manufacturers’  external  commands  to  control  die 
Panasonic  optical  drives  and  the  Workstation  Technologies  video  digitizer  card.  When  authoring 
tools  such  as  these  external  commands  are  available,  they  allow  the  instructional  developer  to 
integrate  and  exchange  a  wide  variety  of  media.  At  this  time,  no  standards  exist  for  exchanging 
multimedia  documents  between  programs  or  across  platforms,  although  an  advisory  committee  is 
developing  a  hypermedia  format  that  could  cross  platforms.  All  of  the  current  programs  have 
proponents,  and  each  user  must  evaluate  them  for  their  own  needs. 

The  ability  to  access  the  many  media  elements  of  a  hypermedia  job  aid  rapidly  is  important, 
but  other  hardware  ami  software  issues  must  be  resolved  before  computer  system  media 
compatibility  is  possible.  Both  Microsoft  Corp.  (developers  of  the  Windows  environment)  and 
Apple  Computer,  Inc.  (developers  of  the  Macintosh  environment)  are  attempting  to  provide  media 
access  to  all  microcomputer  users  while  minimizing  the  need  for  additional  hardware.  ELVIS  uses 
an  analog  video  delivery  system,  with  video  stored  on  WORM  optical  discs.  The  other  common 
option  is  to  use  CD-ROM  optical  media,  which  store  compressed  digitized  video  and  other  types 
of  data.  Development  of  CD-ROM-based  job  aids  can  be  confusing  because  of  the  lack  of 
standards  and  the  varying  software  requirements  to  incorporate  them  into  die  video  workstation 
environment.  Just  as  with  the  analog  optical  disc  used  by  ELVIS,  different  CD-ROM  drives  and 
controllers  require  different  versions  of  device  driver  software  and  compatible  computer  peripheral 
hardware. 

The  microcomputer  industry  is  moving  toward  digital  video,  which  allows  the  video  data 
stream  to  be  compressed  for  storage.  A  number  of  compression  schemes  are  presently  available. 
Whether  Microsoft’s  Video  for  Windows  or  Apple’s  QuickTime  becomes  a  standard  is  immaterial. 
Digital  video  and  audio  communication  architectures,  protocols,  interfaces,  and  media 
compatabilities  must  all  be  brought  into  some  standardization  scheme  that  will  allow  media  aids  to 
be  moved  across  platforms. 

Present  hypermedia  systems  present  major  challenges  to  construction  of  a  job  aid  and 
information  retrieval  by  novices.  Simplified  development  of  a  large,  well-edited  video  database, 
simplified  authoring,  and  easy  retrieval  of  the  many  media  elements  that  constitute  the  databases 
are  needed.  For  example,  the  ELVIS  library  for  visual  data  cataloging  uses  a  commercial  database 
manager  which  the  user  must  master.  There  is  no  provision  to  transfer  cataloged  data  automatically 
into  the  hypermedia  program.  Both  IBM  and  Apple  computers  are  attempting  to  provide  a  “video 
production  studio”  (Donovan,  1991).  Whether  these  extended  capabilities  will  allow  multimedia 
job  aid  development  by  novices  remains  to  be  seen. 

Conclusion 

To  create,  modify,  and  present  a  visual  job  aid  is  difficult  and  requires  a  dedicated  staff  willing 
to  untangle  the  web  of  interrelated  tasks  in  constructing  a  hypermedia  job  aid  system.  Once 
constructed,  users  face  many  challenges  in  browsing  and  seeking  specific  job  information. 
Extensive  efforts,  currently  underway  by  many  major  manufacturers  of  both  hardware  and 
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software,  are  needed  for  the  success  of  hypermedia  in  a  microcomputer  environment.  Whether 
relative  novices  will  be  able  to  operate  die  equipment  needed  to  develop  this  form  of  training  and 
job  aiding  still  is  uncertain. 


Future  Efforts 

1.  Creators  and  users  of  multimedia  job  performance  aids  should  provide  resources,  such  as  a 
software  program  to  introduce  the  mouse  interface  that  students  should  be  required  to  use,  to  guide 
the  user  in  initially  accessing  media. 

2.  Video-based  job  performance  aids  developed  for  use  in  Navy  settings  should  have  an 
in-house  change  advocate  to  ensure  adequate  development  and  use  of  the  aid. 

3.  The  constantly  changing  needs  of  Navy  training  and  task  performance  require  timely 
modification  of  all  media  used  in  job  aids.  Local  Navy  training  and  work  communities  should 
develop  in-house  skills  to  allow  continual  updates  to  computer  media. 

4.  Industry  standards  that  allow  software  and  platform  portability  should  be  monitored  to 
ensure  their  incorporation  into  future  hypermedia  job  aid  development 
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Appendix 

Engineering  Library  Visual  Information  System 
(ELVIS)  Workstation 
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Engineering  Library  Visual  Information  System  (ELVIS)  Workstation 

The  Engineering  Library  Visual  Information  System  (ELVIS)  was  based  on  a  Macintosh  Ilfx 
microcomputer  with  8  megabyte  of  random  access  memory  and  a  300-megabyte  internal  hard  disk. 
For  digital  video  display,  a  Workstation  Technologies,  Inc.  Moonraker  card  produced  30  frames 
per  second  of  16-bit  color  video  on  die  display  monitor  in  a  640  x  480  pixel  window.  To  control 
die  optical  disc  player  horn  the  Macintosh,  a  Hurdler  D  HDS/HQS  extended  serial  board  from 
Creative  Solutions,  Inc.  was  used.  The  optical  disc  machines  consisted  of  one  Panasonic  TQ-303  IF 
Write  Once,  Read  Many  (WORM)  recorder/player  and  one  TQ-3032F  player.  For  display,  a  19- 
inch  Radius  color  monitor  with  Radius  DirectColor/16  video  board  was  used  at  Navy  Personnel 
Research  and  Development  Center  (NAVPERSRANDCEN),  which  Naval  Undersea  Warfare 
Center  (NUWC)  upgraded  to  a  24-bit  color  display  and  video  card.  Sound  was  provided  by  a  small 
pair  of  self-  powered  speakers. 

Hardware  costs  for  the  ELVIS  used  in  this  effort  totaled  approximately  $37,500.  This  included 
both  the  WORM  TQ-303  IF  optical  disc  recorder  and  separate  TQ-3032F  playback-only  device.  A 
student  station  with  a  less  powerful  microcomputer  and  a  play-only  optical  disc  device  would  cost 
approximately  $20,000.  Each  WORM  optical  disc  costs  approximately  $300.  These  costs  excluded 
the  cost  of  video  production  hardware  such  as  cameras,  recorders,  and  signal  stabilizers. 

Authoring  and  Delivery  Software 

A  software  “Manager”  was  developed  to  author,  organize,  access,  and  deliver  job  procedural 
aids.  The  Manager  controls  several  job  aid  functions  including: 

1.  Constructing  and  indexing  a  job  structure. 

2.  Indexing  video  job  structure  steps. 

3.  Entering  text  and  graphics. 

4.  Linking  job  structure  text  to  video  and  graphics. 

5.  Revising  text,  video,  and  graphic  files. 

Thus,  die  Manager  provides  the  author  with  the  ability  to  construct  new  job  structures  with 
media  links,  and  modify  existing  job  structures  by  changing  text,  video,  and  graphics  and  their 
links. 

Three  main  user  interface  windows  include: 

1.  A  set  of  hierarchical  windows  to  present  the  job  procedural  aid  or  job  structure. 

2.  A  video/graphics  window. 

3.  A  “help”  window  for  additional  explanatory  text. 

The  author  selects  the  appropriate  window  position  and  size  for  each  learning  experience.  The 
job  aid  window  displays  job  structure  text.  Ifext  can  be  entered  either  at  the  prompt  or  imported 
from  a  generic  word  processor,  TeachText,  which  is  packaged  with  all  Apple  Macintosh  system 
programs.  Each  text  line  in  this  window  acts  as  a  “command  line.”  By  positioning  the  mouse  over 
any  of  these  lines  any  combination  of  the  following  three  events  can  occur,  depending  on  the 
authoring  links  created:  (1)  opening  of  a  lower  level  job  structure  window,  (2)  display  of  a  video 
clip  or  graphic,  and  (3)  display  of  text  in  the  help  window. 
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The  job  structure  windows  display  successively  more  detailed  job  structure  and  can  be  labeled 
to  represent  the  level  of  detail.  For  example,  if  four  levels  of  windows  are  required,  they  might  be 
labeled  module,  task,  subtask,  and  steps. 

The  video  window  can  display  full  motion,  still  video,  or  graphics.  This  display  is  linked  to  the 
job  structure  window  so  that  media  are  automatically  presented  when  a  selection  is  made  in  the  job 
structure  window.  Whenever  video  is  displayed,  another  window  provides  a  software-based  remote 
control.  By  positioning  the  mouse  cursor  over  die  remote  control  function  desired,  the  optical  disc 
could  be  activated  independently  of  normal  ELVIS  links  within  the  preselected  video  segment. 
Normal  or  speeded  (up  to  10  times  normal  speed)  video  are  available.  In  addition,  stereo  audio  and 
an  on-screen  frame  counter  can  be  turned  cm  and  off.  The  remote  control  proved  to  be  a  valuable 
tool  for  audioring.  Only  one  optical  disc  software  driver  is  provided,  which  effectively  limits 
ELVIS  to  the  Panasonic  drives. 

The  help  window  is  an  auxiliary  text  window  that  can  be  used  for  any  additional  information 
such  as  to  explain  specific  Macintosh  commands.  Students  not  having  sufficient  time  to  familiarize 
themselves  with  the  system  can  refer  to  this  window  for  help.  Tbachlfext  is  required  for  entering 
text  in  this  window. 

The  author  also  has  two  special,  normally  hidden  windows  for  customizing  the  presentation. 
The  “user  preferences”  window  allows  the  author  to  specify  the  number  of  hierarchical  job 
structure  windows,  their  titles,  and  video  window  titles. 

The  “information  packet”  window  allows  specification  of  video,  graphics,  audio  and  text  links. 
The  video  information  subset  requires  the  author  to  select  start/ stop  frames  and  other  information 
such  as  a  wait  state  for  user  input  before  displaying  video. 

The  basic  authoring  procedure,  after  completion  of  all  instructional  development  consists  of  the 
following  steps: 

1.  Catalog  video  and  audio. 

2.  Specify  number  and  titles  of  windows. 

3.  Enter  text  in  appropriate  window. 

4.  Incorporate  graphics,  if  required. 

5.  Link  text,  graphics,  audio,  and  video. 

Following  these  steps  automatically  creates  hyperlinks  between  all  the  media  elements. 


A-2 


Distribution  List 


Distribution: 

Office  of  Chief  of  Naval  Research  (Code  01),  (Code  461) 

Defense  Technical  Information  (DTTC)  (4) 

Copy  to: 

Assistant  for  Training  mid  Personnel  Systems  Technology  (OUSDXA)/R&T  (E&LS) 

Chief  of  Naval  Education  and  Training  (Code  00).  (Code  02),  (Code  L01-B)  (6),  (Code  N32) 
Chief  of  Naval  Operations  (N091),  (N869),  (N889K),  (N911H) 

Chief  of  Naval  Technical  Training  (Code  00)  (2) 

Commander,  Naval  Training  Center,  Great  Lakes,  EL 

Commander,  Naval  Training  Center,  Orlando,  FL 

Commander,  Naval  Training  Center,  San  Diego,  CA 

Commanding  Officer,  Fleet  Training  Center,  San  Diego,  CA  (Code  021.2) 

Commanding  Officer,  Naval  Air  Technical  Training  Center,  Millington,  TN  (AW-A1) 
Commanding  Officer,  Naval  Education  and  Training  Program  Management  Support  Activity 
(Code  03EE1)  (2),  (Code  04)  (2) 

Commanding  Officer,  Naval  Education  Training  Support  Center,  Atlantic 
Commanding  Officer,  Naval  Education  Training  Support  Center,  Pacific 
Director,  Acquisition  Education,  Training  and  Career  Development  Policy,  OUSD  (A) 

Director,  Defense  Activity  for  Nontraditional  Education  Support  (Pensacola,  FL) 

Director,  Training  Systems  Development,  Randolph  AFB,  TX 
Naval  Training  Systems  Center,  Technical  Library  (5) 

Pentagon  Library 


